fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes) - #161
Open
qbq-leo-martens wants to merge 7 commits into
Open
fix(cache): keep the per-turn billing line out of the cached prefix (+3 attribution fixes)#161qbq-leo-martens wants to merge 7 commits into
qbq-leo-martens wants to merge 7 commits into
Conversation
…essages[0] `historyImageSha8` hashed the image blocks on `messages[0]`, described in its own docstring as "the synthetic history message". It is not: whenever a slab anchor exists, `collapseHistory` returns `[...head, syntheticUser, ...tail]` and transform.ts passes `protectedPrefix = slabAnchorIdx + 1`, so on every Claude Code request `messages[0]` is the protected slab message. The field that exists to prove history-image byte-identity was therefore reporting SLAB stability under the history name — a drifting collapse boundary still showed a rock-steady `history_image_sha8` in events.jsonl, which is precisely the signal teamchong#11 attribution relies on. Locate the synthetic message by its banner, the same way `cachePrefixDigest` does. Also split the pinned-prefix digest per layer. `cache_prefix_sha8` proves THAT the cacheable prefix moved but never WHICH layer moved, and the layers fail for different reasons and need different fixes: tools drift when the client loads a deferred tool, system drifts when volatile text (env, git status) rides inside the pinned span, and the imaged head drifts when a collapse boundary or marker placement moves. Emit `cache_prefix_tools_sha8` / `cache_prefix_system_sha8` / `cache_prefix_head_sha8` alongside the aggregate so one session of telemetry names the culprit instead of narrowing it to "somewhere in the prefix". Motivation: a production session showed 530/530 requests with a unique `cache_prefix_sha8` at a constant `cache_prefix_bytes`, 0 cache_read across 87M cache_create tokens — with the aggregate hash alone the cause could not be attributed, and `history_image_sha8` looked stable because it was measuring the wrong message.
…t attribution `cachePrefixDigest` hashed every block up to and including the message that carries pxpipe's imaged prefix. That is not the span Anthropic caches: caching ends at the LAST cache_control marker, and after a history collapse the two differ by construction. The synthetic message's newest freeze chunk re-renders on every turn BY DESIGN (append-only rendering: only completed chunks are frozen) and it sits AFTER the pinned marker — so the boundary-scoped digest changes every single turn even when the cached span is byte-stable, and `cache_prefix_sha8` reads "pxpipe busted its own cache" on healthy traffic. Emit `cache_prefix_marked_sha8` / `_marked_bytes` / `_marker_pos` alongside: the digest through the breakpoint, its size, and where the breakpoint sits as `m<msg>.b<block>`. The marked digest is the one that must hold turn over turn; the marker position makes a roaming breakpoint — a bust cause in its own right, previously invisible — a one-field diagnosis. Tests pin the contract that decides whether cache_read happens at all: two consecutive turns of one session keep the marked span byte-identical while the live tail grows.
… text Claude Code folds `x-anthropic-billing-header: <...>` into the *first* system block, and that line carries a per-turn request id. Anything inside the cache_control-anchored prefix has to be byte-identical across turns, so this one line rewrote the pinned prefix on every single request and cost us the whole prompt cache — every turn paid cache_create instead of cache_read. transform now strips the line out of the system text (leaving no empty block behind) and hands it back on TransformInfo; proxy sets it as the outbound header it always claimed to be. Set PXPIPE_KEEP_BILLING_LINE=1 to fall back to the old in-prompt placement. Also records the two digests that made this diagnosable: a text-only system digest and per-block `index[*]:len:hash4` fingerprints, which is what pinned the churn to block 0 rather than to the collapse boundary.
…xists renderStatsTableFragment read s.eventsWithBaseline, but stats.ts emits eventsWithUsage; the missing field made the comparison false for every session, so the "cache hit (by events)" row silently rendered "-" no matter what the cache did. The type carried the stale name too, so tsc had no chance to catch it. Test pins the value to its own row (the table renders as one line, so a bare toContain passes on a neighbouring number) and fails with '-' against the old expression.
The 100-image limit is a property of the wire, not of pxpipe: it counts
the images the client sent (pasted screenshots, pictures a tool returned)
together with every image we add. We priced only our own, so a request
that arrived already full got imaged further and came back 400/500 —
two sessions became unresumable.
Every imaging path now spends from one shared headroom:
imageHeadroom() = cap - margin - ours - theirs
and the caller's images are counted once, before any rewrite, at both
nesting levels (top-level and inside tool_result).
The gate is quantitative, not boolean. A first cut asked "is there ANY
room left?" and then emitted a whole 15-page slab into it: 94 client
images + a 400k slab put 109 images on the wire. Now:
- the slab must fit whole or not at all — imaging half of it would
re-key the cache prefix on every turn whose client-image count moved;
- each tool_result pages against the LIVE headroom, not a fixed 10;
- after rendering we verify the real page count, because paging is
budgeted at denseGeo.cols while rendering happens at o.cols;
- the history collapse is skipped entirely at zero headroom — its
budget floors at 1, so it would otherwise emit exactly one image
and still fail the request.
Degrading means "do not emit this turn", never "un-emit": each turn is
re-derived from the client's own text transcript, so our images are
never stranded there.
Telemetry: native_images and image_budget_skips reach the event log, and
passthrough_reasons loses an allow-list that swallowed exactly the two
reasons worth diagnosing (kept_sharp, image_budget).
Measured, 300 turns + 60k slab + 60k tool_result:
clients= 0 | slab=3 toolres=3 hist=50 | wire= 56 | text 233k
clients=50 | slab=3 toolres=3 hist=44 | wire=100 | text 206k
clients=80 | slab=3 toolres=3 hist=11 | wire= 97 | text 809k
clients=90 | slab=3 toolres=1 hist= 0 | wire= 94 | text 1053k
Client images always win; we shrink to fit. What remains is a cliff, not
a slope: the collapse is our only reducer and it is image-based, so at a
full wire we drop from compressed straight to raw.
…ge count Three gaps, all of the same kind: the code knew something and never said it. markCacheDead() existed but nothing ever called it. A rejected request never populated a prefix cache, so the append-only freeze it was protecting protects nothing — but only the transform side knew that, and the proxy never told it. responseLeftNoCache() now names the rule and the response path applies it: 413, any 5xx, and a 400 whose body says the prompt is too long. Everything else stays warm on purpose. A 401, a 404, a rate limit say nothing about the cache, and guessing "cold" there would re-cut a live grid and burn the whole prefix as cache_create — the exact cost this module exists to avoid. imageCount is what we RENDERED, and the wire disagrees: the history collapse replaces whole messages, so a tool_result image inside the collapsed range is never sent. Measured on a tool-heavy shape: 91 rendered, 49 on the wire. info.wireImages now counts the outgoing body itself, and the event carries it whenever it differs from the render count. The event log gains history_freeze_step, history_budget_trimmed and history_pack_fill, so the adaptive grid is finally observable: until now nothing recorded whether it had coarsened, why, or whether a repack landed on a cache we believed dead. The dashboard says the same in words, but only when there is something to say: "6 images came from your side and count against the same 100-image request cap · 48 rendered pages never went out — the history collapse absorbed those messages (49 on the wire)". A quiet turn stays quiet. Not fixed here: those 48 absorbed pages are content loss, not just miscounting. An imaged tool_result serializes into the history as "[tool_result]\n[image]", so the tool output survives only as the factsheet's exact identifiers. The cure is ordering — collapse before imaging — which changes the outgoing bytes for every session and re-keys every cache once, so it wants its own change.
The history serializer renders an image block as the literal string `[image]`.
Tool_results were imaged first and collapsed second, so any tool output old
enough to fall into the collapsed prefix reached the model as:
[tool_result]
[image]
The output itself was gone — only the fact sheet's exact identifiers survived.
The slab was already shielded from precisely this ("collapsing it would reduce
slab images to [image] placeholders and destroy the cache anchor"); the same
reasoning was never extended to tool_results.
Swapping the two steps means the collapse serializes the caller's original
text, and only what stays live gets imaged. Measured across three shapes
(200/100/400 turns, a tool_result every 3rd-6th):
chars carried into the history images 21k -> 2062k
pages rendered but never sent 6 -> 0
outgoing text 2226k -> 1712k (-23%)
Less outgoing text is not a rounding artifact: with the tool output imaged
away, the collapse saw a nearly empty transcript and judged collapsing
unprofitable — on the 400-turn shape it did not run at all. Feeding it the
real text lets it do its job.
The block moved verbatim; only its indentation and the two order comments
changed. tests/collapse-order.test.ts pins the property rather than the
ordering (does tool output reach the model at all), and both of its core cases
fail against the previous order.
The wire count now equals the render count, so the disagreement recorded in
tests/native-image-cap.test.ts is gone; that test flips to asserting no page is
paid for and then dropped.
Bytes on the wire change for every session with history, so the first request
after this lands as cache_create once.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Running Claude Code through pxpipe, my prompt-cache hit rate sat at ~0% for a whole
working day while the dashboard cheerfully reported ">95%". That turned out to be two
separate things: one real cache buster, and three places where the telemetry could not
have shown it.
All four are small, independent commits.
1.
fix(cache): the per-turn billing line lived inside the cached prefixClaude Code (>= 2.1.x) folds an
x-anthropic-billing-header: cc_version=...line intoits system text, as an unmarked trailing block. That value is not session-stable —
it carries a per-turn id — so the tail of the cached prefix changed on every single
request: guaranteed full cache miss, every turn, every session.
stripBillingLine()already existed, but it only ever ran on the concatenatedrawSysText, and by then the block Claude Code actually sends had already beenclassified as part of the prefix, so the regex never matched what mattered.
Fix:
liftBillingLineFromSystem()pulls the line out of every system block beforethe static/dynamic split, and the proxy re-attaches it as a real request header
(
x-anthropic-billing-header) — where it belongs, and where it cannot invalidateanything. A block that held nothing but the billing line disappears instead of leaving an
empty shell behind.
Escape hatch:
PXPIPE_KEEP_BILLING_LINE=1restores the old in-prompt placement.Effect on my host: cache reads went from ~0 to the expected 90%+ of input tokens per
session, immediately and across restarts.
2.
fix(cache): bust attribution digested the boundary message, not the MARKED spancachePrefixMarkedSha/cachePrefixMarkedByteswere computed over the boundary messagerather than the span that actually carries the
cache_controlmarker — i.e. the bytesAnthropic really caches. The diagnostics therefore could never point at what changed
inside the pinned prefix (which is exactly what I needed to find #1). Now the digest
covers the marked span and the marker position is recorded alongside it, with per-layer
digests (tools / system / head) so a bust can be attributed to one layer.
3.
fix(cache): history-image hash was read frommessages[0]history_image_sha8was taken frommessages[0]instead of the message that actuallycarries the imaged history, so a perfectly stable history looked like it churned every
turn (and real churn could hide).
4.
fix(dashboard): the by-events cache-hit rate divided by a field that doesn't existrenderStatsTableFragment()dividescacheHitEventsbys.eventsWithBaseline, butstats.tsemits that number aseventsWithUsage.undefined > 0is false, so theevent-based hit rate silently rendered
-forever. Renamed the field inFullStatsSummaryto match the producer.Tests
tests/cache-bust-attribution.test.ts(10 cases), including the regression thatmatters: two turns differing ONLY in the billing id keep a byte-identical cached
prefix, plus per-layer digest movement and marked-span stability while the live tail
grows.
tests/dashboard-api.test.tsandtests/render.test.ts.tsc --noEmitclean,npm run buildclean.Notes
Found while debugging a real ~2h token burn on a self-hosted box; happy to adjust naming,
split the commits differently, or drop #4 into its own PR if you prefer.